Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix ValueType.UNIX_TIMESTAMP conversions #2219

Merged
merged 3 commits into from
Jan 26, 2022
Merged

Fix ValueType.UNIX_TIMESTAMP conversions #2219

merged 3 commits into from
Jan 26, 2022

Conversation

judahrand
Copy link
Member

@judahrand judahrand commented Jan 18, 2022 •
edited
Loading

Signed-off-by: Judah Rand [email protected]

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #2209

Does this PR introduce a user-facing change?:

Fix ValueType.UNIX_TIMESTAMP conversions

@judahrand judahrand requested a review from a team as a code owner January 18, 2022 11:00
@judahrand judahrand requested review from pyalex and removed request for a team January 18, 2022 11:00
adchia
adchia approved these changes Jan 18, 2022
View reviewed changes
Copy link
Collaborator

@adchia adchia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@adchia
Copy link
Collaborator

adchia commented Jan 18, 2022

Is there a way to capture this in a test somehow?

@adchia adchia removed the lgtm label Jan 18, 2022
@codecov-commenter
Copy link

codecov-commenter commented Jan 18, 2022 •
edited
Loading

Codecov Report

Merging #2219 (f919811) into master (62fae05) will decrease coverage by 25.36%.
The diff coverage is 3.70%.

Impacted file tree graph

@@             Coverage Diff             @@
##           master    #2219       +/-   ##
===========================================
- Coverage   84.92%   59.56%   -25.37%     
===========================================
  Files         105      105               
  Lines        8498     8522       +24     
===========================================
- Hits         7217     5076     -2141     
- Misses       1281     3446     +2165
Flag Coverage Δ
integrationtests ?
unittests 59.56% <3.70%> (-0.16%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
sdk/python/feast/type_map.py 38.88% <0.00%> (-27.52%) ⬇️
sdk/python/tests/data/data_creator.py 34.78% <0.00%> (-65.22%) ⬇️
...s/integration/registration/test_universal_types.py 34.96% <10.00%> (-63.75%) ⬇️
.../integration/online_store/test_online_retrieval.py 17.39% <0.00%> (-82.61%) ⬇️
.../integration/online_store/test_universal_online.py 15.20% <0.00%> (-82.03%) ⬇️
...fline_store/test_universal_historical_retrieval.py 19.18% <0.00%> (-80.82%) ⬇️
sdk/python/tests/utils/online_read_write_test.py 20.68% <0.00%> (-79.32%) ⬇️
...gration/registration/test_feature_service_apply.py 31.25% <0.00%> (-68.75%) ⬇️
sdk/python/feast/infra/online_stores/redis.py 26.97% <0.00%> (-67.11%) ⬇️
sdk/python/tests/integration/e2e/test_usage_e2e.py 35.00% <0.00%> (-65.00%) ⬇️
... and 53 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 62fae05...f919811. Read the comment docs.

@judahrand
Copy link
Member Author

Is there a way to capture this in a test somehow?

Hmmm... 🤔 I guess we just need to add a ValueType.UNIX_TIMESTAMP feature to the tests somewhere. I can do this - though in my experience adding new feature types to the tests is a chore 😛

@judahrand
Copy link
Member Author

@adchia Added a datetime Feature to the tests. Hope this is sufficient.

@judahrand judahrand changed the title Handle np.datetime64 to ValueType.UNIX_TIMESTAMP conversion Fix ValueType.UNIX_TIMESTAMP conversions Jan 21, 2022
@judahrand
Copy link
Member Author

@adchia Looks likes there's some bugs in the tests that I don't think I'll be able to fix as I don't have access to the AWS or GCP test infrastructure.

@judahrand
Copy link
Member Author

@adchia I can only imagine that this method is failing to create the table for Redshift:

feast/sdk/python/tests/integration/feature_repos/universal/data_sources/redshift.py

Line 34 in 62fae05

def create_data_source(

Are you able to help debug this as you have access to the Redshift side?

@judahrand judahrand force-pushed the datetime64 branch 2 times, most recently from 090fd73 to 8ca6565 Compare January 21, 2022 18:59
@adchia
Copy link
Collaborator

adchia commented Jan 21, 2022

yeah let me take a look

@judahrand judahrand force-pushed the datetime64 branch 2 times, most recently from 7c40ea3 to fda77d5 Compare January 21, 2022 20:32
@judahrand judahrand requested a review from adchia January 25, 2022 12:41
@judahrand
Copy link
Member Author

@adchia Did you get a chance to look at this at all? Ideally, I think it makes sense for this and #2229 to go in roughly at the same time as there will be merge conflicts so I'm kinda waiting to review that one until this one can move ahead.

@adchia
Copy link
Collaborator

adchia commented Jan 26, 2022

I tried it and couldn't repro (i.e. the tests pass for me locally).

Trying it again now

@judahrand
Copy link
Member Author

I tried it and couldn't repro (i.e. the tests pass for me locally).

Trying it again now

How odd... Maybe something to do with the credentials on the repo Vs local ones?

I think this is a really important fix as currently I'm not sure UNIX_TIMESTAMP features work end to end. Unfortunately, there's not much I can do on my end.

adchia
adchia approved these changes Jan 26, 2022
View reviewed changes
Copy link
Collaborator

@adchia adchia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@feast-ci-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: adchia, judahrand

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@feast-ci-bot feast-ci-bot merged commit ef1884f into feast-dev:master Jan 26, 2022
@felixwang9817
Copy link
Collaborator

cc @judahrand the root cause of the AWS failures on here should be fixed by #2242

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@adchia adchia adchia approved these changes

@pyalex pyalex Awaiting requested review from pyalex pyalex was automatically assigned from feast-dev/maintainers

Assignees

@adchia adchia

@felixwang9817 felixwang9817

Labels
approved kind/bug lgtm ok-to-test release-note size/M
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Serving not working with datetime64 numpy env.
5 participants
@judahrand @adchia @codecov-commenter @feast-ci-bot @felixwang9817